Notes on the Generalisation Performance and Fisher Consistency of Multicategory Classifiers

Existing bounds on the generalisation performance of multicategory classifiers are reviewed and considered in the light of the framework of Hill and Doucet (2005). Insights obtained through the use of this framework are used to further refine these bounds. Similarly, insights into the Fisher consistency of multicategory classifiers which can be obtained from the framework are discussed. 1 Acronyms IID Independent and Identically Distributed SRM Structural Risk Minimisation SVC Support Vector Classification VC Vapnik Chervonenkis 2 Introduction An important aspect of many kernel based algorithms such as Support Vector Classification (SVC), is that Structural Risk Minimisation (SRM) ideas can be applied in order to obtain distribution-free bounds on performance. Such an approach underlies the initial work on SVC in particular, and results in ideas such as the Vapnik Chervonenkis (VC) dimension. In this paper we include a summary of the body of work which is concerned with bounding the performance of multicategory classifiers. This was originally published by Guermeur (2002), but it is important to realise that this paper draws heavily from the work of Elisseeff et al. (1999), to the extent that if one is to fully understand the ideas presented by Guermeur (2002), then it is probably going to be necessary to read it in conjunction with Elisseeff et al. (1999). Further insight is also to be found in work by Paugam-Moisy et al. (2000). New material is presented which is in the same vein, however this is slightly more straightforward and is closer in formulation to the original two-class approach. This material draws on the insights of Hill and Doucet (2005) to reduce the multidimensional bounding problem to a scalar problem, and thus to fully utilise the more traditional approaches to bounding. These approaches are also drawn on by Elisseeff et al. (1999), however by viewing the problem in the manner proposed here it becomes possible to adopt them virtually unchanged. The key references for this work are by Bartlett (1998) and Williamson et al. (2001). Fisher consistency is also examined in this work — it is another desirable classifier property, as has been discussed in the multicategory setting by Terwari and Bartlett (2007); Zhang (2004a,b).